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Preface 


This book is intended to help you diagnose Data Facility Hierarchical Storage 
Manager (DFHSM) problems. It contains internal product information that is 
provided as additional guidance for this diagnosis. The information in this book 
must not be used for programming purposes. 

Because this book is useful only for diagnosing failures in the operation of DFHSM. 
we assume that you have already made a preliminary determination that the 
suspected failure is not a usage error. 


Trademarks 


The following term is a trademark of IBM and is used in this book: 
• MVS/ESA. 


Major Divisions of this Book 

This book is divided into the following sections: 

• Chapter 1, “Introduction” 

• Chapter 2, “Developing and Using the Keywords” explains how to build and use 
a complete keyword string to specify program failures. 

— “Search Argument Procedure” explains how to use the keyword string 
developed from the diagnostic procedures as a search argument in the 
Software Support Facility (SSF), the Early Warning System (EWS), or 
IBMLink/ServiceLink. 

— “Type of Failure Specification” describes the types of program failures for 
which keywords are available. 

• Chapter 3, “Basic Documentation Requirements” lists the minimum 
documentation that will be needed by the IBM support group to diagnose 
customer problems or to document an Authorized Program Analysis Report 
(APAR). 

• Chapter 4, “The Problem Determination Aid (PDA) Facility” explains the use of 
this diagnostic tool. 

• Chapter 5, “Copying PDA Trace, Dump, Journal and DFHSM Log Data Sets to 
Tape” shows JCL examples that demonstrate one way to copy these data sets to 
tape. 

• Chapter 6, “Isolating a DFDSS Error” explains how to obtain a DFDSS dump 
while running DFHSM. 

• Chapter 7, “Locating Modules and Control Blocks in a Dump” shows by example 
how to obtain important information from a dump. 


Preface iii 
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Prerequisite Books 

You should be familiar with the information presented in the following: 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 General 
Information, GH35-0092 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 System 
Programmer’s Guide, SH35-0085. 


Related Books 

The following books contain related information: 

Field Engineering Programming System General Information, G229-2228 

Data Facility Device Support, DADSM and Common VTOC Access Facility: 
Diagnosis Guides, SC26-3968 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 System 
Programmer’s Command Reference, SH35-0083 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 Diagnosis 
Reference Volume 4, LY35-0104 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 Installation 
and Customization Guide, SH35-0084 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 User’s Guide, 
SH35-0093 

Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 Messages 
SH35-0094 ’ 

Resource Access Control Facility (RACF) General Information Manual, 
GC28-0722 

TSO/E Version 2 Programming Guide, SC28-1874 
TSO/E Version 2 Programming Services, SC28-1875. 

MVS/XA Books 

M VS!Extended Architecture Data Administration Guide, GC26-4140 

MVS I Extended Architecture Data Administration: Macro Instruction Reference, 
GC26-4141 

MVSjExtended Architecture Debugging Handbook Volume 1, LC28-1164 

MVS/Extended Architecture Integrated Catalog Administration: Access Method 
Services Reference, GC26-4135 

MVS/Extended Architecture VSAM Administration: Macro Instruction 
Reference, GC26-4152 

MVS/Extended Architecture VSAM Catalog Administration: Access Method 
Services Reference, GC26-4136. 
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MVS/ESA and MVS/DFP Books 

MVS/ESA Application Development Guide, GC28-1821 

MVS/ESA Application Development Macro Reference, GC28-1822 

MVS/ESA Basics of Problem Determination, GC28-1839 

MVS/ESA Diagnosis: System Reference, LY28-1011 

MVS/ESA Diagnosis: Data Areas, Volume 1, LY28-1043 

MVS/ESA Diagnosis: Data Areas, Volume 2, LY28-1044 

MVS/ESA Diagnosis: Data Areas, Volume 3, LY28-1045 

MVS/ESA Diagnosis: Data Areas, Volume 4, LY28-1046 

MVS/ESA Diagnosis: Data Areas, Volume 5, LY28-1047 

MVS/ESA Diagnosis: Special Analysis Techniques, LY28-1840 

MVS/ESA Diagnosis: Using Dumps and Traces, LY28-1843 

MVS/ESA JCL User’s Guide. GC28-1830 

MVS/ESA Message Library: System Codes, GC28-1815 

MVS/ESA Message Library: System Messages, Volume 1, GC28-1812 

MVS/ESA Message Library: System Messages, Volume 2, GC28-1813 

MVS/DFP Version 3 Release 2: System Programming Reference. SC26-4567 

MVS/ESA System Programming Library: Application Development Guide, 
GC28-1852 

MVS/ESA System Programming Library: Application Development Macro 
Reference, GC28-1857 

MVS/ESA System Programming Library: System Management Facilities (SMF), 
GC28-1819 

MVS/DFP Version 3 Release 2: Access Method Services for the Integrated 
Catalog Facility, SC26-4562 

MVS/DFP Version 3 Release 2: Access Method Services for VSAM Catalogs, 
SC26-4570 

MVS/DFP Version 3 Release 2: Diagnosis Guide, LY27-9570 

MVS/DFP Version 3 Release 2: Diagnosis Reference, LY27-9571 

MVS/DFP Version 3 Release 2: General Information, GC26-4552 

MVS/DFP Version 3 Release 2: Macro Instructions for VSAM Data Sets, 
SC26-4569 

MVS/DFP Version 3 Release 2: Macro Instructions for Non-VS AM Data Sets, 
SC26-4558 

MVS/DFP Version 3 Release 2: Managing VSAM Data Sets, SC26-4568 
MVS/DFP Version 3 Release 2: Managing Non-VSAM Data Sets, SC26-4557. 
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Summary of Changes 


Version 2 Release 5.0 

This edition includes the following changes or additions: 

• A chapter has been added that explains the documentation requirements of the 
DFHSM support group. 

• A chapter has been added that explains the Problem Determination Aid Facility 
(PDA). 

• A chapter has been added that demonstrates how to copy PDA trace, dump, 
journal, and DFHSM log data sets to tape. 

• A chapter has been added that explains how to obtain a DFDSS dump while 
running DFHSM. 

• The chapter on locating modules and control blocks in a dump has been 
modified to include more examples. 

• A list of acronyms has been added. 

This edition also includes miscellaneous editorial modifications. 


Version 2 Release 4.0 

This edition includes technical modifications to the information. Change bars are 
incorporated throughout the book to indicate where the technical modifications are 
made. 

Enhancements made within this release are: 

• Recognition and preservation of SMS class names 

• Management of data sets based on management class specifications 

• Conversion of data sets to SMS-managed data sets 

• Expiration of backup versions 

• Support of system reblockable sequential data sets. 

This edition also includes miscellaneous editorial modifications. 


Summary of Changes vii 
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Chapter 1. Introduction 

The purpose of this book is to help you determine if there is a failure in the 
operation of the Data Facility Hierarchical Storage Manager (DFHSM) and how to 
resolve it by the use of keywords. A keyword describes one aspect of a program 
failure. When doing your own software data base search or contacting the IBM 
Support Center for assistance, you should identify your program failure with as 
many of the keywords as apply to the problem. 

The full keyword string describes the following areas: 

• The type of failure 

• The command involved 

• The DFHSM function involved 

• The load module and/or CSECT 

• The component identification number 

• The release and modification level. 

Each keyword that you add makes the search argument more specific. The more 
precise the keyword string, the more selective the search, thereby yielding fewer 
matches in the software data base. If you do not find a similar problem in the data 
base, you can broaden the scope of the search by deleting keywords. 

Software Support Data Bases 

Once a keyword string is selected, it is used to search an online data base. Several 
different software support facilities exist for researching software problems. 

• Software Support Facility (SSF) 

SSF is an IBM online data base containing information about all current 
APARs and PTFs. 

IBM Support Center personnel have access to SSF and are responsible for using 
the set of keywords you provide as a search argument. Support Center 
personnel may help you improve the effectiveness of your search argument. If 
the problem has previously been reported, they can retrieve the records 
describing the failure and correction. 

• IBMLink/ServiceLink 

IBM Link/Service Link is an interactive program that includes applications and 
application functions with the following capabilities: 

Search and browse authorized program analysis reports (APARs) and 
program temporary fixes (PTFs) 

Service request and delivery 

Preventive service planning 

Automatic software alert process. 

• Info/System 

Info/System is an interactive retrieval program product designed for use with the 
companion data base feature Info/MVS. The data base is divided into several 
logical files of related or similar information, one of which is IBM’s Early 
Warning System (EWS). 
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• Early Warning System (EWS) 

EWS is a microfiche copy of the data contained in SSF. It can help you locate a 
problem with the same symptoms as yours and determine a correction to the 
problem. It is organized by component identifier and indexed by APAR system 
code (for example, OY, OZ, UY, and UZ). EWS is published monthly and is 
available to IBM program product customers. 
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, Chapter 2. Developing and Using the Keywords 

I To develop and use a complete keyword string, perform the procedures that follow 

| in “Search Argument Procedure” through “Type of Failure Specification.” 


Search Argument Procedure 

Each keyword describes one aspect of a program failure. The more precisely the 
keyword describes the failure, the more selective the resulting search will be, thus 
increasing the chance of finding an APAR (authorized program analysis report) that 
already addresses a similar failure. 

Use PTF (program temporary fix) numbers as a keyword only if you feel that the 
PTF has caused the problem. The match you are looking for might have been found 
for a program with an earlier or later PTF level than yours. 

You will be most successful in searching a software support data base by observing 
the following rules: 

• Use the keywords you have developed as your search argument. 

• Spell the keywords the same way every time, exactly as they have been specified 
in this manual. 

• Include all the appropriate keywords in any discussion with IBM. If an APAR is 
opened by IBM and documentation is requested, include the keyword strings in 
the APAR documentation you send to IBM. 

• Search the software support data base using the appropriate keywords: 

— Type of failure 

- Command involved 

- DFHSM function involved 1 

- Load module and/or CSECT 
— Component identifier 

- Release and modification level. 


Procedure 

The following procedure explains how to use the keyword string as a search 
argument in SSF, EWS, or IBMLink/ServiceLink. 

1. After entering the keyword string, scan the resulting list of known or similar 
problems. Eliminate APAR fixes that are already installed on your system. 

2. If you find a matching APAR problem description, contact the IBM Support 
Center or use IBMLink/ServiceLink to obtain the fix. 

3. If you do not find a matching APAR problem description, broaden the search 
by eliminating keywords from the search argument as follows: 

a. Omit the release and modification level keyword from the search argument, 
thereby widening the search to include similar failures on other release and 
modification levels. 


| i This refers to basic functions. Use keywords such as backup, recall, etc. 
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I b. Begin dropping other keywords from the search argument to widen the 

I search. 

I 4. If you still do not find a matching APAR problem description, contact the 

I support center. They may be able to refine your search argument further and 

I find a match in the data base. Otherwise they may direct you to gather 

I additional failure-related information, of your call may be referred to the IBM 

I level 2 support group. 


, Specifying the Failing Component Identifier 

I The component identification number should be used whenever DFHSM is 

I suspected of being the failing component. The component identification keyword 

I should be used with other keywords to search SSF, EWS, or IBMLink/ServiceLink. 

I Used alone, it produces a full listing of APARs against DFHSM. 


I Procedure 


I 


The component identification number for DFHSM Version 2 is S66532901. Using 
the release and modification level keyword with the component identification 
number narrows the symptom search to the specified version, release, and 
modification level. The release and modification keyword is explained in “Specifying 
the Release and Modification Level” on page 9 


, Type of Failure Specification 

I The following keywords identify the type of failure that occurred in DFHSM: 

! • ABEND*** 

I • INCORROUT 

I • LOOP 

I • MSG******** 

I • WAIT 

I • ********** (Documentation) 

i • PERFM 

| Select one keyword from the preceding list that best describes the program failure 

I and follow the procedure for the particular failure type. 

I ABENDxxx 

I Use this keyword when the system, DFHSM, or any program that services DFHSM 

| ends abnormally. (Also see Chapter 6, “Isolating a DFDSS Error” on page 21.) 

| When specifying this keyword, you should have available an unformatted 

1 stand-alone dump, an unformatted SYS1.DUMP, or a SYSMDUMP. 

| Do not use this keyword if the abnormal end was forced by the system or the 

| operator because of a prolonged wait state or an endless loop; for those situations, 

| refer to the WAIT and LOOP keywords. 
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I Procedure 

I 1. The format of the keyword string is: 



I Rxxx is the release and modification level keyword, and its optional use can 

I reduce the number of matches in the search. 

| 2. Add the ABEND code to the keyword. For example, use ABEND0C6 if an 

i 0C6 abnormal end occurred. 

| 3. If the abnormal end occurred in a DFHSM module, message ARC0003I 

i indicates the module name and offset of the abnormal end. If message 

I ARC0003I was returned with module and offset unknown, then the abend was 

| not in a DFHSM module and you should use the next procedure. 

I 4. If the address in the program status word (PSW) is not within ARCCTL or 

I ARCWCTL, determine the system service routine that was called or invoked by 

I DFHSM. This can be done by finding the location in storage where the abend 

1 occurred and reading backward in the dump until you find the name of the 

I module that abended. See the following example: 



0ODDCFOO. 00000008 00000000 00000000 00000000 
00DDBFE0. 00000000 00000000 00000000 00000000 
00DDBFF0. 00000000 00000000 00000000 00000000 
00DDC000. 47FOF0OC 001847F0 F006011E 05C041BO 
00DDC010. CFFF4AF0 F00407FF 47F0C080 C9C7C7F0 
00DDC020. F1F9C2E9 F0F261F1 F961F8F8 C8C4D7F2 
00DDC030. F2F3F040 E4E8F1F8 F6F4F840 F5F6F6F5 
00DDC040. 60E7C1F2 404DC35D 40C3D6D7 E8D9C9C7 
00DDC050. C8E340C9 C2D440C3 D6D9D74B 40F1F9F8 
00DDC060. F26B40F1 F9F8F240 03C9C3C5 D5E2C5C4 
00DDC070. 40D4C1E3 C5D9C9C1 D3E24060 40D7D9D6 
00DDC080. D7C5D9E3 E840D6C6 40C9C2D4 40009180 
00DDC090. 10254780 C114900F D0004120 20005F20 


. 00 .... 00 . 

...00....0{ .IGG0 
19BZ02/19/88HDP2 
230 UY18648 5665 
-X2A (C) C0PYRIG 
HT IBM C0RP. 198 
2, 1982 LICENSED 
MATERIALS - PRO 
PERTY OF IBM .j. 
-A...}. 



I Note: This example was obtained by using IPCS. 

I In this example, an abend occurred at address xxxxxxxx, which is not shown. 

| Reading backward in the dump, the module name IGG019BZ was found. This 

| is the module that abended. There can be a few lines or several pages between 

| the abend address and the associated module name. 

| After determining the module name, the format of the keyword string should be 

| ABENDxxx modulename where modulename is the non-DFHSM module name 

| just determined. Do not use the DFHSM component identification number 

| (566532901) or the Rxxx keyword. 

| Note: Further assistance can be obtained by looking at the diagnosis guide for 

| the product that contains the non-DFHSM module. 
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User Exit Abends 

Note: Because IBM does not support user exits, it is the user's responsibility to 
debug user exit abends. 

If the system fails in a user exit, you can force DFHSM to generate a dump. You 
can prevent DFHSM from continuing its normal ESTAE recovery process by using 
the DFHSM PATCH command to set the MCVTFDMP flag on. For example: 

PATCH .MCVT.+2D X'Gl* VERIFY(.MCVT.+2D X'GO 1 ) 

You should maintain the other flags, if any, from the current setting in the byte being 
patched. Use the DISPLAY command to determine the current setting of the other 
flags before issuing the PATCH command. 

See Data Facility Hierarchical Storage Manager: Version 2 Release 5.0 Diagnosis 
Reference Volume 4 for more information about the PATCH and DISPLAY 
commands. 


INCORROUT 

Use this keyword when the expected output was not received (missing) or when the 
output was different from what was expected (incorrect). 


Procedure 


1. The format of the keyword string is: 


566532901 Rxxx INCORROUT keyword 


where keyword is a word or words relating to what you were doing at the time 
incorrect output was generated. Rxxx is the release and modification level 
keyword, and its optional use can reduce the number of matches in the search. 

2. If you are receiving incorrect output and using a command such as HLIST, you 
should put the keyword HLIST (or whatever command you are using) in the 
search string. 


LOOP 

Use this keyword if some part of the program code runs endlessly; include situations 
in which some part of the output repeats endlessly. 

Do not use this keyword for an intentional loop used to wait for some resource. For 
this condition refer to the WAIT keyword. 

Procedure 

1. The format of the keyword string is: 


J 66532901 Rxxx LOOP modulename 


Rxxx is the release and modification level keyword, and its optional use can 
reduce the number of matches in the search. 
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2. If a DFHSM program suspends activity for no apparent reason, it could be in a 
loop or wait state. A loop can be evident from a never-ending request found by 
the QUERY ACTIVE command or from some other symptom such as a page of 
output being printed repeatedly. If a loop is not evident, use the WAIT 
keyword. 

3. If you have a complete storage dump, locate the address where the LOOP 
occurred. DFHSM modules contain the module name near the beginning of the 
module. See the following example: 


00007C30. 

00000000 

00000000 

00000000 

00000000 



00007C40. 

00000000 

00000000 

00000000 

00000000 



00007C50. 

00000000 

00000000 

00000000 

00000000 



00007C60. 

47F0F022 

1DC1D9C3 

C1C2D4E2 

C74BF8F7 

.00..ARCABMSG.87 

- 

00007C70. 

F0F4F44B 

F1FOF0F0 

F3F14BC8 

C8D4F2F3 

044.100031.HHM23 


00007C80. 

FOF290EC 

D00C18CF 

1FFF43F0 

C5081F00 

02..}.0E... 


00007C90. 

BF07C509 

47F0C03C 

00000200 

89F00008 

. .E. .0}.10.. 


Note: 

This example was obtained by using IPCS. 



i 4. Locate the system trace table to determine if a DFHSM task is being dispatched 

I repeatedly in the same module. For an example of a system trace table, see 

i MVS/Extended Architecture Debugging Handbook Volume 1. If the loop 

I involves more than one DFHSM module, the repetitive invoking of these 

i modules by DFHSM can be identified by the PDA Trace Facility. See 

1 Chapter 4, “The Problem Determination Aid (PDA) Facility" on page 13 for 

1 further information. 


I MSGxxxxxxxx 


j 


i 


Use this keyword when any of the following conditions occur: 

• DFHSM issued a message that indicated an internal program error. 

• A message is not issued in some set of conditions that should cause it to be 
issued. 

• A message is issued in some set of conditions that should not cause it to be 
issued. 

• A message contains invalid or missing data. 


i Procedure 


1. The format of the keyword string is: 


566532901 Rxxx MSGxxxxxxxx 


| Rxxx is the release and modification level keyword, and its optional use can 

! reduce the number of matches in the search. 

! 2. Replace the xxxxxxxx in the MSG.*x;c;«:.xx>c keyword with the message 

! identifier. For example, if the message number is ARC0025I, the 

| MSGxxxxjccxx keyword is MSGARC0025I. 
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l WAIT 

! Use this keyword when the following condition exists: 

i • While waiting for some condition to be satisfied, the system, DFHSM, or some 

I program that services DFHSM has suspended activity without issuing a message. 

I Do not use this keyword if the wait occurs after an abnormal end. as the result of an 

! unanswered message, or because of an endless loop in DFHSM. Use the ABEND 

! or LOOP keyword. 

: Procedure 


1. The format of the keyword string is: 



! Rxxx is the release and modification level keyword, and its optional use can 

I reduce the number of matches in the search. The module name and resource 

! name are optional keywords. 

I 2. The wait state can have many causes. Listed below are some of the most 

I common of these causes: 

I * The system resource manager has marked DFHSM non-dispatchable 

i because of excessive paging. 

I • DFHSM has serialized on a resource and is locked out by another task in 

I this system or by a task in another system. 

i • A backup task is waiting for a backup volume to become available. These 

! backup volumes were made unavailable because another processing unit 

! went down while running backup. 

j xxxxxxxxxx (Documentation) 

i Use the documentation keyword when a programming problem appears to be caused 

i by incorrect, missing, or ambiguous information in one of the DFHSM books. 

i Procedure 


1. The format of the keyword string is: 



Keyword is an optional word, such as the command or DFHSM function 
involved. For example, if the DFHSM function was migration, use the keyword 
migration. 

2. Enter the order number of the publication in place of xxxxxxxxxx, the 
documentation keyword. Include the letter prefix and version number, but omit 
all hyphens. For example, if the order number is SH35-0083-02, the 
Documentation keyword is SH35008302. 

3. Locate the page in the book on which the error or omission occurs and prepare 
a description of the problem. You should also include this information in the 
error description when submitting documentation for an APAR. 
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i PERFM 


Use this keyword when some part of DFHSM performs below explicitly stated 
expectations and the performance problem cannot be corrected by system tuning. 
Most performance problems can be related to system tuning and should be handled 
by system engineers and system programmers. 


I Procedure 

i 


1. The format of the keyword string is: 


566532901 Rxxx PERFM 


Rxxx is the release and modification level keyword, and its optional use can 
reduce the number of matches in the search. 


i Specifying the Release and Modification Level 

i This keyword identifies the release and modification level of DFHSM. RS02 

I represents Release 5, Modification Level 0. 

I The following table shows the release and modification keyword for the current 

: releases of DFHSM Version 2. 


Release 

and 

Modification 

Keyword 

Description 

240 

R402 

base 


R411 

English panels 


R412 

Kanji panels 

250 

R502 

base 


R511 

English/ISPF panels 


R513 

English/ISMF panels 


Because it is difficult to determine exactly which release and modification level to 
specify, first try all the symptom-related keywords to conduct your search. If the list 
of matches is too large, add the release and modification level keyword to reduce the 
number of matches. 

The release and modification keyword is required on APAR forms. 
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Chapter 3. Basic Documentation Requirements 


Documentation Needed for Problem Diagnosis 

When the proper diagnostic procedures have been followed, user specifications have 
been checked for accuracy, and the keyword search has proven unsuccessful, you 
should then contact the IBM Support Center for further assistance. 

If the Support Center representatives cannot immediately resolve the problem, they 
may pass the problem call to the DFHSM Level 2 support group. When the support 
group representatives phone, they will request that you have certain documentation 
at hand to aid in the diagnosis of the problem. 

The documentation normally needed to engage in a telephone conversation with the 
level 2 support group will be: 

• The PDA trace, preferably in a form that can be browsed on line 

• An unformatted dump, obtained as a SYS 1.DUMP or SYSMDUMP 

The dump should be available to be browsed on line by IPCS or as a printed 
hardcopy. 

Because it is possible that the support group will need other documentation, either to 
pursue problem diagnosis or to open an APAR, you should keep the following for 
possible future use: 

• The system console log from around the time the problem occurred 

• The portion of the journal pertaining to the problem 

• Portions of the DFHSM X and Y logs pertaining to the problem 

• The command activity log, if appropriate 

• The specific activity log, if appropriate. 

These should be retained until you are sure they will not be needed. 


Documentation Needed for an Authorized Program Analysis Report 
(APAR) 

If IBM opens an APAR addressing your problem, you should be prepared to supply 
such information as the customer number, version, release and modification level, 
current maintenance level (from PTF list), and the keyword string used to search 
SSF, EWS, or IBMLink/ServiceLink. 

You will also be asked to supply various types of information that describe the 
DFHSM functions used, data base, environment, or activities. Applicable items of 
information from the following list may also be requested. 

• JCL listings (for a failure related to a BATCH job). 

• System console log. 

• Portions of the DFHSM X and Y logs pertaining to the problem. 
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• The command activity log, if appropriate. 

• The specific activity log, if appropriate. 

• PDA trace. 

• If the problem relates to a specific data set, a DFDSS logical dump of the data 
set on tape. 

• A copy of any traces or dumps taken. 

• For a wait failure, a complete description of the resource being waited for and 
the program module that is waiting. 

• For LOOP failures, some indication of the location of the loop. This could be 
taken from console trace addresses or it could be one or more module names 
taken from the PDA trace. Try to get at least a partial trace of the loop. 

• For documentation failures, the location of the error in the manual and a 
description of the problem it caused. 

• For PERFM failures, a description of the actual performance, the expected 
performance, and the source of the performance specification. 

Note: When you submit any of the requested documentation on tape, you should 
write it on a STANDARD LABEL TAPE. A hardcopy of the DCB information for 
each data set and the JCL used to create the tape will also be required. 
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Chapter 4. The Problem Determination Aid (PDA) Facility 

During DFHSM processing, the Problem Determination Aid Facility gathers 
diagnostic information about DFHSM’s processing, stores this information in a 
circular file 2 within storage, and periodically writes it to a circular file on DASD. 
The circular DASD file consists of two data sets, ARCPDOX and ARCPDOY. 
ARCPDOX is the active data set. 

For detailed information on using the Problem Determination Aid, see 

• “DFHSM Problem Determination Aid Facility” in Data Facility Hierarchical 
Storage Manager: Version 2 Release 5.0 Installation and Customization Guide 

• “Tracing DFHSM’s Processing” in Data Facility Hierarchical Storage Manager: 
Version 2 Release 5.0 System Programmer’s Guide. 

Browsing the PDA Data Set 

The following examples, obtained by using the ISPF browse function and the HEX 
DATA command, are given to aid you in identifying the trace entries when you are 
discussing the PDA trace data set with the IBM level 2 support group. 

The following example shows a portion (four entries) of a PDA trace data set as it 
would appear on your terminal screen, just before issuing the HEX DATA 
command. 

Note: The file name (in this example DFHSM.PDOX) is set by you in your startup 
procedure. 


— 

BROWSE — DFHSM.PDOX .. LINE 

COMMAND »»•> HEX DATA 

LOCK .ENQ ..D....ARCGPA ..BAKQ..E..L. 

UNLK .DEQ ..D....ARCGPA ..BAKQ..L. 

MCTL .ENTR ..D.N.Y. 


IRCB .ENTR . .D.U.H.MCTL 


2 This type of file appends data until full; then, starting at the beginning of the file, subsequent incoming data 
overwrites the data already there. 
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| The following example shows the same data set as in the previous example, after 

I issuing the HEX DATA command. Note that the display has expanded and each 

! entry is now shown in EBCDIC and hexadecimal. 


BROWSE — DFHSM.PDOX--LINE 

COMMAND =«■> 

LOCK .ENQ ..0....ARCGPA ..BAKQ..E..L. 

D3D6C3D24005CSD5D840008CEEC470040A29C1O9C3C7O7C1404006FFC2C1D2D803FFC503FFD30000 

0800080000800000000000000000090800000000 

UNLK .DEO ..D....ARCGPA ..8AKQ..L.. 

E4D5D3D24008C4C5D840008CEEC470080A28C1D9C3C7D7C1404006FFC2CIO2D803FFD30000000000 

0000800000009008080008000800908800808080 

MCTL .ENTR ..D.N.Y. 

D4C3E3D34000CSD5E3D9808D3EC471D50BFC80302F709E00FDBDA800 

0000800080000808000800800800000880098000 

IRCB .ENTR . .D.U.H.MCTL 

C9D9C3C24081C5D5E3D9O08D3EC471E40CFC00302EC800D4C3E3D340 

0000880008008008008000000080000000000000 


i For simplicity, the remaining examples show only the first entry. 

! Module Name Identifier 

I In the following example, the highlighted five bytes are the last five bytes of the 

I DFHSM module that generated the entry. The highlighting is done in both the 

I EBCDIC and hexadecimal-portions of the examples, whenever it is significant to do 


BROWSE - DFHSM.PDOX . LINE 

COMMAND —> 

HfflS -ENQ ..D....ARCGPA ..BAKQ..E..L. 

MMJMM 05C5D5D84BO88CEEC47OO4OA29C1D9C3C7D7C14O4OO6FFC2C1D2D883FFC583FFD3888O 

0008000000800000800080008008080080008000 


! Trace Point Identifier 

I The highlighted byte in the following example is the individual trace point identifier. 


-\ 

BROWSE - 0FHSM.PD0X . LINE 

COMMAND »■»> 

LOCK .ENQ ..D....ARCGPA ..BAKQ..E..L. 

D3D6C3D240f3 C5D5O840008CEEC470040A29C1D9C3C7D7C14O4OO6FFC2C1D2D803FFC5O3FFD30000 
0000000000000800000000000008080000000000 
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I Type-of-trace Identifier 

I The highlighted four bytes in the following example show the type-of-trace identifier. 

I 

I 

I 


I 


I Task Control Block (TCB) Identifier 

I The highlighted three bytes in the following example show the unique TCB identifier. 

| The first byte is the address space identifier. If this byte is zeroed, the TCB is in 

| DFHSM primary address space; otherwise it is in secondary address space. The next 

I two bytes are the second and third bytes (middle two bytes) of the TCB address. 

I 

I 

I 


I Time of Day Clock Entry 

I The highlighted three bytes in the following example show the fourth, fifth, and sixth 

I bytes of the eight-byte processing unit time-of-day (TOD) clock entry. 

i 
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I Data Segment Length 

I A number of variable length data segments can follow the TOD clock entry, one for 

I each trace value recorded. In the following example there are four data segments. 

I The highlighted byte shows the length of the first segment (OA or ten bytes), and the 

I bracketed area is the segment. Note that the length byte is included in the total 

I length. 



I Parameter Keyword Identifier 

I The highlighted byte in the first data segment is the parameter keyword identifier, as 

! shown in the following example. 



I Parameter Data 

! The highlighted bytes in the following example show the variable data for this entry. 


i 

i 


BROWSE — OFHSM.PDOX - LINE 

COWAND =*«> 


LOCK .ENQ ..O....EfflaZ! • .BAKQ..E 

D3D6C3D24005C5D5D840008CEEC470040A29_ 

0000088008000000000000080000800010088000 


C1D9C3C7D7C14040 


O6FFC2C1D2D803FFC503FFD30000 


I This bracketed area is the second data segment of this entry and the highlighted byte 

I is the length of the segment (six bytes). 
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I Padding 

I 

I 


The highlighted area in the following example shows that the last entry is optionally 
padded with binary zeros. 


I 


'l 

BROWSE — DFHSH.PDOX -—. LINE 

COMMAND »»=» 

LOCK .ENQ ..0....ARCGPA ..BAKQ..E..L. 

O3D6C3DZ40O5C5D5D84OO08CEEC470O40A29C1O9C3C7O7C14O4006FFC2C1D2D803FFC5O3FFD3EH 


oooooooooooooooooooooooooooooooooooooooo 
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Chapter 5. Copying PDA Trace, Dump, Journal and DFHSM 
Log Data Sets to Tape 


The JCL examples in this section show how to copy certain data sets onto a tape so 
that they can be submitted to IBM as APAR documentation. The tapes that you 
use for this purpose must be standard label tapes. 

Note: Because of numerous variations in hardware and software, these JCL jobs 
may not work in all environments. They are shown only as examples. Also note 
that you should choose the data set names, relative file numbers, tape volume serial 
number, and unit type (shown highlighted) that are consistent with your naming 
conventions. 

When you create your JCL, do not use high-level qualifiers of SYS1, HSM, or 
DFHSM in the data set names specified in the SYSUT2 DD cards, as these may be 
reserved names in the system where your tape will be off-loaded. It is recommended 
that you use the APAR number as the high-level qualifier, followed by a qualifier 
that describes the content of the data set. 


PDA Trace Data Set 

This JCL shows a method of copying the PDA trace data set to tape. 


//PDACOPY JOB MSGCLASS=A 
//SI EXEC PGM=IEBGENER 
//SYSPRINT DD SYS0UT=A 
//SYSIN DD DUMMY 

//SYSUT1 OD DSN=DFHSM.PDOY,DISP=SHR 

//SYSUT2 DD DSN=0Y99999.PDOY,LABEL=(1,SL),VOL=SER=TAPE01, 
// DISP=(NEW,KEEP),UNIT=TAPE 


Dump Data Set 

This JCL shows a method of copying a dump data set to tape. 


//DMPCOPY JOB MSGCLASS=A 
//SI EXEC PGM=IEBGENER 
//SYSPRINT DD SYS0UT=A 
//SYSIN DD DUMMY 

//SYSUT1 DD DSN-SYS1.DUMP86,DIS P=SHR 

//SYSUT2 DD DSN=0Y99999.DUMP,LABEL=(2,SL),VOL=SER=TAPE01, 

// DISP=(NEW,KEEP),UNIT=TAPE 


Journal Data Sets 

If you are running SETSYS CDSVERSIONBACKUP(TAPE), then copy the 
appropriate journal data set(s) to tape. Be sure when you send them to IBM to 
include a hardcopy of the JCL used. 
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! If you have allocated the CDSs in your startup procedure, use the following JCL to 

i copy the online journal data sets to tape. 


I Note: if DFHSM 

! //JRNCOPY JOB MSGCLASS=A i-is in operation, the 

i //SI EXEC PGM=IEBGENER . disposition for DFHSM 

i //SYSPRINT DO SYS0UT=A must also be SHR. 

//SYSIN DO DUMMY j 

! //SYSUT1 DD DSN=DFHSM.JRNL.BACKUP.V6008002,DISP=SHR 

i //SYSUT2 DD DSN=0Y99999.JRNL,LABEL=(3,SL),V0L=SER=TAPE01, 

I // DISP=(NEW,KEEP),UNIT=TAPE 


I DFHSM Log Data Set 

i This JCL shows a method of copying the DFHSM log Y data set to tape. 

i If the data that is needed is on the active DFHSM log (log X), run the JCL job, 

i issue the SWAPLOG command, and run the JCL job again. 


//LOGCOPY JOB MSGCLASS=A 
//SI EXEC PGM=IEBGENER 
//SYSPRINT DD SYS0UT=A 
//SYSIN DD DUMMY 

//SYSUT1 DD DSN=DFHSM.LOGY,DISP=SHR 

//SYSUT2 DD DSN=0Y99999.LOGY,LABEL=(4,SL),VOL=SER=TAPE01, 

// DISP=(NEW,KEEP),UNIT=TAPE 


Activity Log Data Sets 

This JCL shows a method of copying the activity log data sets to tape. 

If you are running ACTLOGTYPE(DASD) and you wish to process the current 
activity logs, issue the following: 

• SETSYS ACTLOGTYPE(SYSOUT) 

This closes and de-allocates the current activity logs. 

• SETSYS ACTLOGTYPE(DASD) 

This allocates new activity logs. Now the old logs can be browsed or copied to 
tape using the following JCL example. 


//ALOGCOPY JOB MSGCLASS=A 
//SI EXEC PGM=IEBGENER 
//SYSPRINT DD SYS0UT=A 
//SYSIN DD DUMMY 

//SYSUT1 DD DSN=HSMACT.HI.BAKLOG.089012.T085721,DISP=SHR 
//SYSUT2 DD DSN=0Y99999.ACTLOG,LABEL=(5,SL),V0L=SER=TAPE01, 
// DISP=(NEW,KEEP),UNIT=TAPE 
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Chapter 6. Isolating a DFDSS Error 

While running DFHSM and using DFDSS as a data mover, you can encounter a 
maintenance problem if an error occurs that is not isolated to DFDSS. You can try 
to duplicate this error by running DFDSS separately. If this fails to produce similar 
results, DFHSM can be patched to cause DFDSS to generate a dump at the time 
this specific error occurs. 

Patch the DFHSM field MCVTDSSM as follows: 

PATCH .MCVT.+454 nnn 

For nnn, substitute the 3-character number from the DFDSS message (ADRmwix) 
that was issued when the problem first occurred. 

When a message number is patched into this normally blank field, DFDSS will 
produce either SYSUDUMP or SYSABEND dump output that can help to 
determine the cause of the error. 
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Chapter 7. Locating Modules and Control Blocks in a Dump 

Many important DFHSM data areas can be located by their addresses in the 
Management Communication Vector Table (MCVT). Figure 1 shows how two base 
items (MCVT, ARCESD) can be located when the dump includes low storage. 

Note: The following information and examples refer only to the DFHSM primary 
address space. For dumps occurring in the ABARS secondary address space, 
contact the IBM Support Group for further assistance. 


Queue 

Control 



Figure 1. Locating the MCVT and Module ARCESD 


When the low storage is not available, it is helpful to know that MCVT follows the 
module ARCCBS, and that ARCESD and ARCCBS are ordered as the first two 
modules of the DFHSM load module ARCCTL. 

The first occurrence of the word MCVT will be in ARCESD (see the following 
example), and the associated address will give the actual location of the MCVT. 

All DFHSM modules and major DFHSM control blocks, such as the MCVT, the 
management control record (MCR), and the backup control record (BCR) are each 
pointed to by the ARCESD module. 

Note: Dump examples were obtained and processed by using IPCS. 

The following example, a segment of a dump, shows the beginning of ARCESD. It 
contains eight-byte module name fields followed by corresponding four-byte address 
fields for each DFHSM module. By using this information, you can efficiently locate 
these modules within the dump. Note that even though most modules have their 
names at the beginning of the module, ARCESD does not. 
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Beginning of ARCESD 


O0O060A0. 

000060B0. 

000060C0. 

000060D0. 

000060E0. 

000060F0. 

00006100. 

00006110. 

00006120. 

00006130. 

00006140. 

00006150. 

00006160. 

00006170. 

00006180. 


00000000 

0000774C 

C4C3D940 

40404040 

0000774C 

C2E2D9E2 

C1404040 

00007C60 

C1D9C3C1 

C4E5C440 

0O0090E0 

C1D9C3C1 

D3D5C4E2 

0000BCA8 

C1D9C3C1 


00000000 

C2C3D940 

40404040 

000O74B4 

D4E2D9E2 

C1404040 

00211B88 

C1D9C3C1 

C4C5E7E3 

00008D28 

C1D9C3C1 

D3D5C4D9 

0000AC30 

C1D9C3C1 

D3E2C4D7 


1 

D4C3E5E3 
40404040 
0000768C 
C1D9C3C3 
Cl404040 
00189128 
C1D9C3C1 
C3D9C5D3 
00000000 
C1D9C3C1 
D3C9E2E3 
0000A5A8 
C109C3C1 
0306D3C4 
0000D4F8 


40404040 

0000757C 

D4C3D940 

E5E3404O 

00186328 

D6E2D9E2 

C2D4E2C7 

00008170 

C1D9C3C1 

D3C3C2E5 

00O0A368 

C1D9C3C1 

D3D6C740 

0000CC58 

C1D9C3C1 


.HCVT 

...<BCR ...0 

OCR ....HCR 
....ARCCVT 
...<MSRSA .... 
BSRSA ..j.OSRS 

A ...hARCABMSG 
..0-ARCACREL..a. 
ARCADEXT....ARCA 
OVD ....ARCALCBV 
...\ARCALIST..t. 
ARCALNDR..vyARCA 
LNDS....ARCALOG 
...yARCALOLD.... 
ARCALSDP..M8ARCA 


The following example demonstrates how to locate a specific module (in this case 
ARCALOG) in the dump. The module name and its corresponding address are 
located in ARCESD, and then the leftmost column is scanned until this address is 
again located. 


008060A0. 

000060B0. 

O0006OC0. 

O00060DO. 

000060E0. 

000060FO. 

00006100. 

00006110. 

00006120. 

00006130. 

00006140. 

00006150. 

00006160. 

00006170. 

00006180. 


00000000 

000O774C 

C4C3D940 

40404040 

0000774C 

C2E2D9E2 

C14O4040 

00007C60 

C1D9C3C1 

C4E5C440 

000090EO 

C1D9C3C1 

D3D5C4E2 

8888BCA8- 

C1D9C3C1 


00000000 
C2C3D940 
40404040 
00007484 
D4E209E2 
Cl404040 
00211B88 
C1D9C3C1 
C4C5E7E3 
O0008D28 
C1D9C3C1 
D3D5C4D9 
00AC30 
-] D9C3C1 
E2C4D7 


D4C3E5E3 
40404040 
0000768C 
C1D9C3C3 
C1404040 
00189128 
C1D9C3C1 
C3D9C5D3 
00008000 
C1D9C3C1 
D3C9E2E3 
0000A5A8 
C1D9C3C1 
D3D6D3C4 
000004F8 


40404040 

000O757C 

D4C3D940 

E5E34040 

00186328 

D6E209E2 

C2D4E2C7 

00008170 

C1D9C3C1 

D3C3C2E5 

0000A368 

C1D9C3C1 

D3D6C748 

0000CC58 

C1D9C3C1 


.MCVT 

...<BCR ...@ 

OCR ....HCR 
....ARCCVT 
...<MSRSA .... 
BSRSA ..j.OSRS 
A ...hARCABMSG 
.,0-ARCACREL..a. 
ARCADEXT....ARCA 
DVD ....ARCALCBV 
...\ARCALIST..t. 
ARCALNDR..vyARCA 
LNDS....ARCALOG 
...yARCALOLD.... 
ARCALSDP..M8ARCA 


• ♦ I 

0000BCA8. 

0000BCB0. 

0000BCCO. 

0000BCD0. 


Address x'BCA8' points to the 
beginning of ARCALOG. Note that APAR 

n numbers, if included, appear at the 
beginning of modules. 

BCA8 

♦ 


C1D3D6C7 404BFBF7 
F3F94BD6 E8F0F8F3 
1FFF43F0 CFAC1F00 


47F8F822 1DC1D9C3 
F2F4F74B F1F1F4F5 
F3F190EC D00C18CF 
BF07CFAD 47F0C0C3 


.08..ARC 
AL0G .87247.1145 
39.OY08331..}... 
...0 . 0 {. 


The MCVT, in addition to storing flags, also contains pointers of work queues and 
chained lists. Figure 2 on page 25 shows some of the useful data sources. 

In addition to DFHSM data areas, information normally formatted in a dump is 
useful. This includes registers from the save areas at entry to ABEND and the list 
of outstanding enqueues. The registers point to the automatic storage areas 
associated with the last routines in control. Almost all of the useful debugging 
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information is contained in the automatic storage for each routine, in the registers at 
the time an abnormal condition occurred, or in DFHSM data areas pointed to from 
the MCVT. 


Management 
Communication 
Vector Table (MCVT) 


MCVTQCT 


4(4) 


Queue 

Control Management Work Element (MWE) 

Table (QCT) In Common Service Area 


12(0 


MCVTMWEH 


36(24) 


MCVTMVQH 


140(8C) 


MCVTL2CR 


232(E8) 


MCVTBUQH 


284(11C) 


MCVTVSQH 


352(160) 


MCVTDRSP 


IT 


360(168) 


MCVTSMQH 


1012(3F4) 


MQCTMWEH H 


MWEFWD 


MWEFWD 


General HWE Queue 


MWEFWD 

m 






Operator MWE Queue 


MCVTOPQH 


-► 

MWEFWD 

■ 


52(34) 




II 



Recall MWE Queue 


MCVTRSQH 

IE 

-► 

MWEFWD 

S 


60(30 




II 



Migration MWE Queue 


MCVTMGQH 


-► 

MWEFWD 

m 


68(44) 




ii 



Mounted Volume Table (MVT) (Non-SMS-Managed Volumes) 


MCVTFWD 

■ 

MCVTFWD 

m 



II 





Level 2 Control Record (L2CR) 


Backup/Recovery MWE Queue 


MWEFWD 

■ 



1 



Volume Statistic Records (VSRs) 


VSRFWO 

m 

VSRFWD 

m 



mm 





Daily Statistic Record (DSR) 


Mounted Volume Table (MVT) (SMS-Managed Volumes) 


MVTFWD 


MVTFWD 


IT 


Figure 2. Using the Management Communication Vector Table 
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List of Acronyms 

ABEND Abnormal end. 

APAR Authorized program analysis report. 

BCDS backup control data set. 

A VSAM, key-sequenced data set that contains 
information about backup versions of data sets, 
backup volumes, dump volumes, and volumes 
under control of the backup and dump 
functions of DFHSM. 

BCR Backup control record. 

CSECT Control section. 

CVT Communications vector table. 

DADSM Direct Access Device Space Management. 

DASD direct access storage device. 

A device in which the access time is effectively 
independent of the location of data. 

DFDSS Data Facility Data Set Services. 

An IBM licensed program used to copy, move, 
dump, and restore data sets and volumes. 

DFHSM Data Facility Hierarchical Storage Manager. 

An IBM licensed program used to manage 
volumes and data sets in an MVS operating 
environment. 

EBCDIC Extended Binary Coded Decimal Interchange 

Code. 

EWS Early Warning System. 

IPCS Interactive problem control '"stem. 

ISPF Interactive System Productivity Facility. 

An IBM licensed program used to develop, test, 
and run application programs interactively. 
ISPF is the interactive access method for all 
storage management functions. 

JCL Job control language. 

MCR Management control record. 

MCVT Management communication vector table. 

MVS/ESA MVS/Enterprise Systems Architecture. 

An MVS operating system environment that 
supports accessing of virtual storage in multiple 
address spaces and data spaces. 

PDA Problem determination aid. 

PDSE Partitioned data set extended. 

RACF Resource Access Control Facility. 

An IBM-licensed program that provides access 
control by identifying and verifying users to the 
system. RACF authorizes access to resources, 
logs unauthorized access attempts, and logs 
accesses to protected data sets. 

SDSP small data set packing. 

In DFHSM, the process used to migrate data 
sets that contain equal to or less than a 
specified amount of actual data. The data sets 
are written as one or more records into a 
VSAM data set on a migration level 1 volume. 


SMS Storage Management Subsystem. 

An operating environment that helps automate 
- and centralize the management of storage. To 

manage storage, SMS provides the storage 

administrator with control over data class, 
storage class, management class, storage group, 
and ACS routine definitions. 

SSF Software Support Facility. 

TCB Task control block. 

TOD Time of day. 

TSO time sharing option. 

An option on the operating system for a 
System/370 that provides interactive time 
sharing from remote terminals. 

TTOC Tape table of contents record. 

VSAM virtual storage access method. 

An access method for indexed or sequential 
processing of fixed and variable-length records 
on direct access devices. The records in a 
VSAM data set or file can be organized in 
logical sequence by a key field (key sequence), 
in the physical sequence in which they are 
written on the data set or file (entry-sequence), 
or by relative-record number. 

VTOC volume table of contents. 

(1) A table on a direct access volume that 
describes each data set on the volume. 

(2) An area on a disk or diskette that describes 
the location, size, and other characteristics of 
each file and library on the disk or diskette. 
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Index 


A 

abends, user exit 6 

abnormal end (ABEND) failure 4 

acronyms, list of 27 

activity log data set, copying to tape 20 

C 

copying to tape 

activity log data sets 20 
DFHSM log data set 20 
dump data set 19 
journal data set 19 
PDA trace data set 19 

D 

data bases, software support 

EWS (Early Warning System) 2 
IBMLink/ServiceLink 1 
Info/System 1 

SSF (Software Support Facility) 1 
data set, copying to tape 
activity log 20 
DFHSM log 20 
dump 19 
journal 19 
PDA trace 19 

developing a keyword string 3 
DFDSS dump, producing 21 
DFDSS error, isolating 21 
DFHSM log data set, copying to tape 20 
documentation failure 8 
documentation requirements 
for APARS 11 
for problem diagnosis 11 
dump data set, copying to tape 19 

E 

Early Warning System (EWS) 

See software support data bases 
examples 
JCL 19 

locating modules and control blocks in a dump 23 

F 

failing component identifier, specifying 4 
form of keyword string for 
ABENDxxx 5 
INCORROUT 6 
LOOP 6 
MSGxxxxxxxx 7 


form of keyword string for (continued) 

PERFM 9 
WAIT 8 

xxxxxxxxxx (documentation) 8 

I 

1BM Link/ServiceLink 

See software support data bases 
incorrect output (INCORROUT) failure 6 
Info/System 

See software support data bases 
introduction 1 
isolating a DFDSS error 21 

J 

JCL examples 19 

journal data set, copying to tape 19 

K 

keyword 

ABENDxxx (abnormal end) 4 
INCORROUT (incorrect output) 6 
LOOP 6, 12 

MSGxxxxxxxx (message) 7 
PERFM (performance) 9 
WAIT 8 
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